We extend the standard theory of cosmological perturbations to homogeneous but anisotropic universes. We present an exhaustive computation for the case of a Bianchi I model, with a residual isotropy between two spatial dimensions, which is undergoing complete isotropization at the onset of inflation; we also show how the computation can be further extended to more general backgrounds. In presence of a single inflaton field, there are three physical perturbations (precisely as in the isotropic case), which are obtained (i) by removing gauge and nondynamical degrees of freedom, and (ii) by finding the combinations of the remaining modes in terms of which the quadratic action of the perturbations is canonical. The three perturbations, which later in the isotropic regime become a scalar mode and two tensor polarizations (gravitational wave), are coupled to each other already at the linearized level during the anisotropic phase. This generates nonvanishing correlations between different modes of the CMB anisotropies, which can be particularly relevant at large scales (and, potentially, be related to the large scale anomalies in the WMAP data). As an example, we compute the spectrum of the perturbations in this Bianchi I geometry, assuming that the inflaton is in a slow roll regime also in the anisotropic phase. For this simple set-up, fixing the initial conditions for the perturbations appears more difficult than in the standard case, and additional assumptions seem to be needed to provide predictions for the CMB anisotropies.